Skip to content

fix: close provider gaps after the AI SDK 7 upgrade - #251

Merged
grabbou merged 8 commits into
mainfrom
fix/apple-sdk-7-gaps
Oct 24, 2026
Merged

grabbou merged 8 commits into
mainfrom
fix/apple-sdk-7-gaps

Conversation

@grabbou

@grabbou grabbou commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Closes the gaps left in the providers after the AI SDK 7 upgrade: Apple reported 0 tokens for every call, countTokens wasn't reachable from the model, Apple streams dropped tool calls on iOS 26 and threw for structured output, and the provider factories used pre-v7 names. Each commit can be reviewed on its own.

Commits

  1. refactor(apple): export the native module as AppleFoundationModels. AppleFoundationModels.ts was added in feat: graceful apple context window error handling, countTokens API #212 only to guard countTokens on older native builds, and refactor(apple): drop redundant native method guards #248 removed that guard. The TurboModule is now exported directly, like the other native modules and as 0.12.0 did. Also removes the unused src/stream.ts.
  2. feat(apple): report token usage from foundation models.
    • iOS 27: passes through Response.usage / Snapshot.usage (input, cached input, output and reasoning tokens), for the on-device model and Private Cloud Compute.
    • iOS 26.4: counts the transcript before and after the response with the on-device tokenizer.
    • Otherwise: usage is undefined instead of 0.
  3. fix(core-ai, llama): report unknown token usage as undefined. Same change for the other providers.
  4. feat(apple, llama): count tokens through language models. apple().countTokens(text) and llama.languageModel(...).countTokens(text). @react-native-ai/utils already calls model.countTokens for history summarization and fell back to length / 4 because no provider defined it. AppleFoundationModels.countTokens stays.
  5. feat(apple): stream tool calls on ios 26 and structured output.
    • iOS 26 snapshots carry no transcript entries, so tool calls and results are now read from session.transcript while the response streams.
    • streamText with Output.object no longer throws. Apple's partial JSON snapshots aren't prefixes of the final JSON (key order changes too), so the provider sends the complete object once generation finishes.
  6. refactor: align provider factories with ProviderV4 naming.
    • Every factory sets specificationVersion: 'v4'.
    • embeddingModel (apple, llama) and rerankingModel (llama) added; the published textEmbeddingModel / rerankModel stay as deprecated aliases.
    • core-ai drops textEmbeddingModel, which was never published.
    • mlc gets createMlcProvider(); mlc.languageModel() keeps working.
    • Docs, README and skills use the new names.

Breaking change

Native generateText now resolves { content, usage } instead of an array of parts, so AppleFoundationModels.generateText returns that object. The docs example is updated.

Testing

  • bun lint, bun run typecheck, and package tests: apple 19 (6 new, including streamText + Output.object through the real SDK), llama 35, core-ai 5.
  • AppleLLMImpl.swift typechecks against the iOS 27 device and simulator SDKs with Swift 6.4, so the #if compiler(>=6.4) branches are compiled.
  • React Native codegen parses the updated NativeAppleLLM spec.
  • A FoundationModels script on macOS 27 confirmed the native behavior this relies on:
    • Native usage reported 64 input / 13 output tokens; the tokenizer fallback estimated 56 / 14.
    • session.transcript holds the tool call and output entries (same id) before the first text chunk streams.
    • Partial JSON snapshots aren't prefixes of the final JSON.

Not tested:

  • The example app wasn't built or run, so the AppleLLM.mm change hasn't been compiled.
  • The iOS 26 streaming path wasn't run on an iOS 26 runtime.

🤖 Generated with Claude Code

grabbou and others added 6 commits October 24, 2026 23:43
AppleFoundationModels.ts was added in #212 to reject countTokens on native
builds without it. #248 removed that guard, so the wrapper only re-declared
the native methods. Export the TurboModule directly, like the other native
modules in the package and as 0.12.0 did.

Also remove src/stream.ts, an unused stream helper still typed with
LanguageModelV2StreamPart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Responses and streams reported 0 input and 0 output tokens, so AI SDK
usage, prepareStep thresholds and telemetry always saw zero.

- iOS 27 passes through the usage Foundation Models measures (input,
  cached input, output and reasoning tokens), for the on-device model and
  Private Cloud Compute
- iOS 26.4 counts the transcript before and after the response with the
  on-device model's tokenizer
- otherwise usage is undefined instead of 0

Native generateText now resolves { content, usage } and onStreamComplete
carries usage, which changes the shape returned by
AppleFoundationModels.generateText.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core-ai always reported 0 tokens and llama fell back to 0 when llama.rn
returned no timings. AI SDK treats 0 as a measured count, so totals looked
real. Report undefined instead, as adk does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@react-native-ai/utils summarizes history by calling countTokens on the
model it is given, but no provider defined it, so it always fell back to
four characters per token.

Apple language models count with the on-device model's tokenizer
(iOS 26.4+) and llama.rn models with their own tokenizer.
AppleFoundationModels.countTokens stays available.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On iOS 26, stream snapshots have no transcript entries, so streamText never
reported Apple tool calls. Read them from the session transcript instead,
which already holds the tool call and output entries when the first text
chunk arrives.

streamText with Output.object threw "Streaming JSON responses is not yet
supported." Native streams now send the snapshot's JSON. Partial snapshots
are not prefixes of the final JSON (even key order changes), so the provider
sends the complete object once the stream finishes.

Schema streams also report reasoning and tool entries on iOS 27.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI SDK 7 names provider factories embeddingModel and rerankingModel and
gives providers a specificationVersion.

- every provider factory sets specificationVersion: 'v4'
- apple and llama add embeddingModel, and llama adds rerankingModel; the
  published textEmbeddingModel and rerankModel stay as deprecated aliases
- core-ai drops textEmbeddingModel, which was never published
- mlc adds createMlcProvider, so mlc is a callable provider like the others;
  mlc.languageModel() keeps working

Docs, README and skills use the new names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ai Ready Ready Preview Oct 24, 2026 8:05pm UTC

Request Review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Llama token counting loses its instance receiver in history summarization, silently falling back to estimates.

1 open issue
What changed in this PR

This PR closes provider gaps after the AI SDK 7 upgrade, improving token accounting, Apple streaming, and provider API consistency.

Changes:

  • Adds Apple usage reporting and model-level token counting; represents unknown usage as undefined.
  • Supports Apple structured-output streaming and iOS 26 tool-call reporting, while simplifying native exports.
  • Aligns provider factories with v4 naming and updates documentation and tests.
File Description
website/​src/​docs/​llama/​reranking.md Uses rerankingModel in examples.
website/​src/​docs/​llama/​model-management.md Documents embeddingModel.
website/​src/​docs/​llama/​getting-started.mdx Updates the embedding factory name.
website/​src/​docs/​llama/​embeddings.md Updates embedding examples.
website/​src/​docs/​apple/​generating.md Documents streaming, usage, and native return changes.
website/​src/​docs/​apple/​embeddings.md Uses embeddingModel in examples.
skills/​react-native-ai/​references/​llama-provider.md Updates embedding and reranking factory names.
skills/​react-native-ai/​references/​apple-provider.md Updates the embedding factory name.
README.md Updates provider API examples.
packages/​utils/​README.md Describes summarizer token counting.
packages/​mlc/​src/​index.ts Exports createMlcProvider.
packages/​mlc/​src/​ai-sdk.ts Adds a callable v4 provider factory.
packages/​llama/​src/​ai-sdk.ts Adds token counting, unknown usage, and v4 factory names.
packages/​core-ai/​src/​ai-sdk.ts Aligns factory naming and unknown usage.
packages/​apple-llm/​src/​stream.ts Removes the unused streaming helper.
packages/​apple-llm/​src/​NativeAppleLLM.ts Defines native usage and generation result types.
packages/​apple-llm/​src/​index.ts Exports the native module directly.
packages/​apple-llm/​src/​AppleFoundationModels.ts Removes the redundant native wrapper.
packages/​apple-llm/​src/​ai-sdk.ts Maps usage and adds token counting and structured streaming.
packages/​apple-llm/​src/​__tests__/​ai-sdk.test.ts Tests usage, structured streaming, and token counting.
packages/​apple-llm/​ios/​AppleLLMImpl.swift Adds native usage and streaming transcript handling.
packages/​apple-llm/​ios/​AppleLLM.mm Forwards usage in stream completion events.
packages/​adk/​src/​ai-sdk.ts Declares the v4 provider specification.

Review effort: Balanced

Comment thread packages/llama/src/ai-sdk.ts Resolved
grabbou and others added 2 commits October 25, 2026 00:04
countPromptTokens took countTokens off the model and called it without a
receiver. The llama.rn model reads this.context in countTokens, so it threw
and the summarizer silently fell back to four characters per token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
countTokens on a Private Cloud Compute model counts with the on-device
model's tokenizer, which may not match the server model, and needs the
on-device model to be available. Calling it an approximation claimed more
than is known.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@grabbou
grabbou merged commit 02e3ede into main Oct 24, 2026
9 checks passed

This branch was successfully deployed

1 active deployment
Preview — d5062c17 Deployed Oct 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants